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LONG-TERM  GOAL 

The  long-term  goal  of  this  research  is  to  construct  the  Navy’s  next  generation  NWP  model  using  new 
numerical  methods  specifically  tailored  to  take  full  advantage  of  distributed-memory  computers.  In 
order  to  take  full  advantage  of  the  new  computer  architectures  the  spherical  global  domain  must  be 
partitioned  into  local  sub-domains,  or  elements,  which  can  then  be  solved  independently  on  the 
multiple  processors  of  these  computers.  The  numerical  methods  used  on  these  sub-domains  must  be 
not  only  local  in  nature  but  also  high-order  in  accuracy  and  highly  efficient.  Thus  the  final  objective  of 
this  project  is  to  construct  a  new  NWP  model  which  is  as  accurate  as  the  current  spectral  model  while 
much  more  efficient  thereby  allowing  for  finer  resolution  forecasts. 

OBJECTIVES 

The  objectives  of  this  project  are  to  construct  high-order  local  methods  for  the  Navy’s  next  generation 
NWP  model.  The  high-order  accuracy  of  these  methods  will  ensure  that  the  new  model  yields  the 
same  accuracy  as  the  current  spectral  model  while  the  locality  of  these  methods  will  ensure  that  the 
efficiency  of  the  new  model  increases. 

APPROACH 

To  meet  our  objectives  we  explore  the  three  local  high-order  methods: 

1.  Continuous  spectral  element  in  space  with  an  explicit  Eulerian  scheme  in  time  (C-SEM), 

2.  Discontinuous  spectral  element  in  space  with  an  explicit  Eulerian  scheme  in  time  (D-SEM), 

3.  Spectral  element  in  space  with  a  semi-implicit  semi-Lagrangian  scheme  in  time  (SESL). 

These  methods  will  be  used  to  discretize  the  horizontal  operators.  For  this  reason  we  propose  to 
implement  these  methods  to  the  spherical  shallow  water  equations  as  these  equations  contain  all  of  the 
horizontal  operators  required  in  a  full  3D  NWP  model.  The  power  of  continuous  and  discontinuous 
spectral  element  methods  is  that  they  are  high  order  accurate,  like  spectral  methods,  yet  are  completely 
local  in  nature  -  meaning  that  the  equations  are  solved  independently  within  each  individual  element 
and  processor.  The  reason  for  exploring  discontinuous  spectral  element  methods  is  they  are  more  local 
than  continuous  methods  thereby  resulting  in  a  potentially  more  efficient  NWP  model  on  distributed- 
memory  computers.  In  addition,  discontinuous  methods  are  not  only  globally  but  also  locally 
conservative  which  translates  to  proper  handling  of  large  gradient  phenomena  such  as  fronts  and 
hurricanes.  Semi-Lagrangian  methods  are  also  being  considered  because  these  methods,  like  high 
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order  methods,  have  minimal  dispersion  errors.  This  property  is  important  for  properly  capturing 
atmospheric  phenomena.  In  addition,  semi-Lagrangian  methods  offer  vast  improvements  in 
computational  efficiency  due  to  the  longer  integration  time  steps  that  they  permit. 


Upon  validation  of  the  three  horizontal  discretization  methods,  the  vertical  integration  scheme  for  the 
full  3D  NWP  model  will  be  included.  The  final  stage  for  the  dynamics  is  to  include  performance¬ 
enhancing  features  such  as  Helmholtz  solvers  for  the  semi-implicit  semi-Lagrangian  method,  as  well 
as  Message-Passing  Interface  (MPI)  and  Open  MP  implementations. 

WORK  COMPLETED 

The  three  horizontal  discretization  methods  (C-SEM,  D-SEM,  and  SESL)  have  been  applied  to  the 
shallow  water  equations  on  the  sphere.  These  methods  have  been  tested  extensively  and  three  papers 
have  been  written  concerning  these  innovative  approaches  (Ref.  [1,2,3]).  The  new  spectral  element  3D 
NWP  model  is  undergoing  extensive  validation.  Four  test  cases  have  been  applied  to  the  new  model 
and  this  work  was  presented  at  a  conference  in  Montreal  (Ref.  [4]). 

RESULTS 

Shallow  Water  Equations.  To  show  the  accuracy  of  the  three  horizontal  discretization  methods  we 
show  results  for  the  methods  for  nonlinear  zonal  geostrophic  flow  (Case  2  in  Ref.  [1]).  Figure  1  shows 
the  spectral  convergence  of  the  3  methods.  It  should  be  mentioned  that  while  C-SEM  and  D-SEM, 
which  are  Eulerian  in  time,  use  a  Courant-Friedrichs-Levy  (CFL)  restricted  time  step,  SESL  uses  a 
time- step  10  times  larger. 


Figure  1:  L2  error  (y-axis)for  the  continuous  spectral  element  method  (C-SEM),  discontinuous 
spectral  element  method  (D-SEM),  and  spectral  element  semi-Lagrangian  method  (SESL)  as  a 

function  of  basis  function  order  N  (x-axis). 
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Note  the  high-order  accuracy  achieved  by  all  three  methods.  In  fact,  all  three  methods  converge  at  the 
same  rate  as  the  spectral  method.  This  result  confirms  that  all  three  methods  are  capable  of  yielding 
results  equal  to  or  better  than  the  current  spectral  model. 


In  addition  to  SESL  allowing  larger  time  steps  it  has  been  proven  that  the  error  can  in  fact  decrease 
with  increasing  time  step.  The  error  of  SESL  is  bounded  as  follows 
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where  K  corresponds  to  the  order  of  integration  of  the  particle  trajectory  equation 

dx  .  . 

—  =  u(x,t) 
dt 


and  N  corresponds  to  the  order  of  interpolation  and  hence  the  order  of  the  spectral  element  basis 
functions.  The  most  amazing  thing  about  this  order  of  accuracy  result  is  that  as  we  increase  N  (as  is 
done  in  the  spectral  element  method)  we  can  increase  the  time  step  without  incurring  any  time 
discretization  errors  provided  that  K  is  sufficiently  high.  Figure  2  shows  the  L2  error  versus  time  step 
for  advection  on  the  sphere  for  N=16  and  K=2  and  4. 


A  t  (in  days) 


Figure  2:  L2  error  (y -axis)  for  the  spectral  element  semi-Lagrangian  method  (SESL) 
with  N=16  as  a  function  of  time-step  (x-axis)  for  K=2  and  K=4. 


We  can  see  that  for  K=2  the  error  increases  linearly  with  time  step  after  0.005  days.  By  increasing  K  to 
4  we  see  that  the  error  remains  flat  even  for  time-steps  of  0.5  days.  The  point  of  this  study  is  to  show 
that  the  spectral  element  semi-Lagrangian  method  (SESL)  will  not  only  yield  exponentially  accurate 
solutions  but  it  can  do  so  using  very  large  time  steps  thereby  offering  increased  accuracy  and 
efficiency  over  any  other  numerical  method. 
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3D  NWP  Equations.  As  a  first  test  to  validate  the  new  3D  NWP  model,  a  still,  isothermal  atmosphere 
was  run  for  a  100  day  integration.  The  results,  as  expected,  was  that  very  little  happened.  This  test  was 
used  to  check  the  stability  of  the  algorithms  and  to  ensure  that  no  “numerical  sources”  arise. 

As  a  second  test,  we  used  Case  1  with  a  conical  mountain  embedded  along  the  Equator  and  ran  the 
code  for  100  days.  The  model  showed  completely  symmetric  flow  patterns  thereby  confirming  that  no 
biasing  is  introduced  by  either  the  numerical  scheme  or  the  grid. 

As  a  third  test,  we  ran  Case  2  with  the  mountain  located  at  30N  degrees  latitude.  Figure  3  shows  the 
results  at  the  surface  for  N=10  with  L=10  vertical  levels  after  100  days. 


3D  Velocity  Vectors 


Figure  3:  Velocity  vectors  for  flow  over  a  mountain  with  L=N=10  as  a  function  of  longitude 
(lambda)  and  latitude  (theta)  at  the  surface  for  a  100  day  integration. 


As  a  final  case,  we  ran  the  Held-Suarez  test  case.  This  test  case  is  the  primary  test  run  by  all  new 
atmospheric  models  because  it  mimics  a  realistic  atmosphere  with  full  physics.  We  ran  this  case  for 
100  days  and  measured  conservation  of  mass  and  energy.  Figure  4  shows  the  contour  plot  of  the  mean 
zonally-averaged  zonal  velocity  as  functions  of  latitude  and  vertical  coordinates  for  N=  4  and  F=20 
vertical  levels. 


4 


3D  U  Velocity 


90S  60S  30S  EQ  30N  60N  90N 


Figure  4:  Zonally-averaged  zonal  velocity  (U)  for  the  Held-Suarez  test  case  with  N=4,  L=20  as  a 
function  of  latitude  (theta)  and  vertical  coordinate  (sigma)  for  a  100  day  integration. 


Figure  4  compares  well  with  the  results  from  Held  and  Suarez  and  clearly  shows  the  formation  of  the 
jet  stream. 

IMPACT 

The  Navy  Operational  Global  Atmospheric  Prediction  System  (NOGAPS)  is  run  operationally  by  the 
Fleet  Numerical  Meteorology  and  Oceanography  Center  (FNMOC)  and  is  the  heart  of  the  Navy’s 
operational  weather  prediction  support  to  nearly  all  DOD  users  worldwide.  Our  work  here  targets  the 
next  generation  of  this  system  for  the  next  generation  of  computer  architectures.  These  architectures 
are  expected  to  be  distributed  memoiy,  commodity  based  systems  with  enormous  theoretical 
computational  power.  However,  exploiting  this  capability  will  require  drastically  redesigning  many 
important  model  algorithms. 

TRANSITIONS 

Improved  algorithms  for  model  processes  will  be  transitioned  to  the  6.4  Large-Scale  Atmospheric 
Models  project  (PE  0603207N,  CAPT  Clark,  SPA  WAR,  Project  Manager)  as  they  are  ready,  and  will 
ultimately  be  transitioned  to  FNMOC  with  future  NOGAPS  upgrades. 

RELATED  PROJECTS 

Some  of  the  technology  developed  for  this  project  will  be  used  immediately  to  improve  the  current 
spectral  formulation  of  NOGAPS  in  conjunction  with  the  6.2  project:  Improved  Subgrid-scale  Physical 
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Parameterizations  for  Global  Modeling  Global  (NRL  BE-35-2-18).  In  addition,  the  work  herein  will 
complement  the  work  done  on  constructing  an  efficient  message  passing  NOGAPS  as  is  being  done  in 
the  6.3  project:  Development  of  Two  Complete  Numerical  Weather  Prediction  Systems  for 
Heterogeneous  Scalable  Computing  Environments  (PE  0603704N). 

SUMMARY 

We  have  constructed  a  semi-implicit  spectral  element  semi-Lagrangian  (SESL)  and  discontinuous 
spectral  element  methods  for  the  horizontal  operators.  In  addition  we  have  continued  to  validate  the  3D 
spectral  element  NWP  model.  The  results  of  this  year  have  shown  the  feasibility  of  our  approach  and 
confirm  that  the  long-term  goals  of  this  project  will  be  realized  on  schedule. 
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